Repository navigation
Add a sweeper for Terraform state locks orphaned by killed workers - #797
Closed
emmanuelmaduwuba-lyft wants to merge 2 commits into
Closed
emmanuelmaduwuba-lyft wants to merge 2 commits into
emmanuelmaduwuba-lyft wants to merge 2 commits into
Conversation
A worker that is killed mid-run leaves the S3 .tflock object and DynamoDB lock item behind, and every later plan for that root fails with a 412 / ConditionalCheckFailedException until someone deletes them by hand. Add a cron-scheduled Temporal workflow that lists the locks and deletes the ones that are provably orphaned: older than a threshold, held by the Atlantis worker, a plan lock, and with no open Terraform workflow for the root in Temporal. Apply locks and locks held by anyone else only alert. It is off unless lock_sweeper.enabled is set and runs in dry-run mode by default.
emmanuelmaduwuba-lyft
force-pushed
the
terraform-lock-sweeper-pr
branch
from
October 19, 2026 17:28
be53230 to
6efebac
Compare
The tests and a doc comment used real internal repo and root names. Replace them with example names; the sweeper only parses <prefix>/<root>/terraform.tfstate.tflock, so behavior is unchanged.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
A worker that is killed mid-run (SIGKILL after the 60s SIGTERM window, a pod kill, an OOM) leaves the Terraform state lock behind: the S3
<root>/terraform.tfstate.tflockobject and the DynamoDB lock item. Every later plan for that root fails with412 PreconditionFailed/ConditionalCheckFailedExceptionuntil someone deletes both by hand. A recent rollout left locks on dozens of roots that sat for about 18 hours.Change
server/neptune/lyft/locksweeper: a cron-scheduled Temporal workflow (every 30 minutes, fixed workflow ID so one schedule exists cluster-wide) that lists the locks in both stores and, for each one:min_age_minutes(default 120, config floor 60, the longest a Terraform activity runs) or if Temporal has an openTerraformworkflow for the root. If the Temporal check fails the lock is skipped.lock_sweepersection to the global config, wired into the deploy worker. It is off unlessenabled: trueand runs in dry-run mode unlessdry_run: false.Impact
No behavior change unless
lock_sweeper.enabledis set; a disabled section converts to the zero value so existing config is unaffected. When enabled in dry-run it only logs and emits metrics. Existing workflows and activities are unchanged. To roll out, add thelock_sweepersection inatlantis-private(needs a submodule bump) withowner_prefixset to the identity Atlantis's workers take locks as. It is required when enabled and has no default, so automatic deletes only ever apply to those locks, run in dry-run on staging first, then setdry_run: false. Rollback is removing the section or settingenabled: false.Testing
go testpasses forserver/neptune/lyft/locksweeper,server/config/...,server/neptune/temporalworker/...andcmd/...in a clean clone ofmainat 74bb20f. The full./server/neptune/...suite also passed before the owner-prefix check was added; after it I re-ran only the packages listed. New tests cover the decision rules, key parsing, the delete guards (non-lock keys, md5 item, changed ETag, failed DynamoDB condition), query injection, the workflow including dry-run and the owner check, and config defaults and validation.WorkflowType/atlantis_repository/atlantis_rootquery; that the Atlantis role can list, get and delete the lock objects and scan and delete from the lock table; and anything against real AWS or Temporal. The S3 delete cannot be conditional with the v1 SDK, so a lock replaced between the ETag check and the delete would still be removed.